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Abstract Body 


Background / Context: 

Despite years of implementing standards-based accountability systems, many teachers today lack 
the necessary preparation to develop standards-based instructional strategies and to inform 
decisions utilizing student assessments (Drake, 2007). The eMINTS {enhancing Missouri’s 
Instructional Networked Teaching Strategies) National Center at the University of Missouri 
(UM), in partnership with the Missouri Department of Higher Education (DHE), offers the 
eMINTS professional development (PD) program to teams of educators, especially those serving 
high-need students. eMINTS PD generates building wide reform by helping teachers master the 
translation of any state standards and information from assessments into engaging classroom 
practices that employ technology. The program is based on four underlying research-based 
components: inquiry-based learning, high-quality lesson design, community of learners, and 
technology integration, and addresses issues identified as barriers in the consistent use of 
standards-based instruction and technology. The program provides teachers with more than 250 
hours of PD spanning two years and support that includes monthly classroom visits. As part of 
refining and improving eMINTS PD, eMINTS staff integrated the Intel® Teach Elements 
Program recently, adding a third year of PD to help teachers sustain and build on the first two 
years of eMINTS PD. The third year combines online and face to face training for teachers to 
expand their use of project-based learning. Teachers also gain access to Intel’s suite of online 
tools designed to involve students in 21st century higher-order thinking and problem solving. 

A decade of evaluation of the eMINTS original two-year program consistently has shown 
promise in changing teachers’ practice and raising student achievement. However, these 
evaluation studies focused on intermediate elementary students (Grades 3-6), and were either 
non-equivalent group or pre-post designs with no comparison groups. In 2010, the eMINTS 
National Center received an i3 (Investing in Innovation) validation grant for a randomized 
control trial to evaluate the efficacy of the eMINTS PD program in increasing seventh- and 
eighth-grade students’ mathematics and English language arts (ELA) achievement, 2U‘ century 
skills and academic engagement. By using a rigorous evaluation design and by focusing on 
middle school grades, this study aims to expand the knowledge base about the effectiveness of 
the eMINTS’ program. 

Purpose / Objective / Research Question / Focus of Study: 

The current report focuses on the impact after the second year of implementation of the eMINTS 
program.^ It addresses the following research questions: 

1 . What is the impact of the eMINTS Comprehensive Program on the performance of 
Grades 7 and 8 students in mathematics and communication arts? 


* The final study report (after the third year of implementation) will also assess the impact the third year of eMINTS 
PD using the Intel Teach Program Thinking with Technology (eMINTS+Intel Teach), added to the original two-year 
eMINTS PD program. 
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2. What is the impact of the eMINTS Comprehensive Program on the 21st century skills, 
including communication, technology literacy, and critical thinking, of Grades 7 and 8 
students? 

3. What is the impact of the eMINTS Comprehensive Program on the academic engagement 
of Grades 7 and 8 students? 

Setting: 

The study focused on grade 7 and 8 students in 60 high-poverty rural Missouri schools. 

Population / Participants / Subjects: 

Each of the 60 study schools was randomly assigned to one of three groups: (1) the traditional 
eMINTS Comprehensive Program, (2) the eMINTS Comprehensive Program plus a third year of 
professional development using Intel Teach Program courses, or (3) business as usual. ^ 
Randomization was blocked by the grade range in each school. Of the 60 schools, 30 schools 
serve Grades PK-8 or K-8 (Block 1), eight schools serve Grades 6-8 or 7-8 (Block 2), and the 
remaining 22 schools serve Grades 6-12 or 7-12 (Block 3). Enrollment in the study schools was 
fewer than 200 students, reflecting the small and rural aspects of the sample schools. Across both 
eMINTS and control schools, about 5 percent of students were minorities, 58 percent qualified for 
free or reduced-priced lunch, less than 2 percent were English language learners, and between 12 
percent and 13 percent had an identified disability. The mathematics and communication arts 
analytic samples (RQl) each consisted of about 3,000 students, and the analytic samples for 21st 
century skills and student engagement (RQ2 and RQ3) each had about 2,500 students. 

Intervention / Program / Practice: 

The eMINTS Comprehensive Program consists of professional development for school 
principals, district and school technology coordinators, and classroom teachers. Teachers receive 
about 240 hours of professional development across two years in which they learn to design 
high-quality inquiry-based lesson plans, implement inquiry-based learning strategies, build 
community among teachers and students, and integrate technology into classroom instruction. 
Before the start of the school year, a certified eMINTS instructional specialist (elS) is assigned to 
a collection of schools — according to his or her geographic location — to provide formal and 
individualized professional development to teachers and communicate with principals and 
technology coordinators. eISs facilitate the ongoing development of a school -based leadership 
team within each school to support implementation and ensure that the required technology 
infrastructure and equipment functionality are maintained in eMINTS classrooms. During the 
first year of professional development, teachers receive 126 hours of formal professional 
development in 26 sessions that are held throughout the school year. At the end of the first year, 
teachers spend up to 12 additional hours developing a classroom website. The Year 2 curriculum 


^ Two treatment schools dropped out before the first year of implementation, and one control school closed after the 
first year of implementation 

^ Note that although the schools were randomized into three groups because of the addition of Intel Teach to the 
two-year eMINTS program in Year 3, for the Year 2 analysis, which is the focus of this report, there are effectively 
only two groups: (1) Groups 1 and 2 combined (eMINTS); and (2) Group 3 (Control). 
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focuses on classroom management, website enhancement, assessment, interdisciplinary teaching 
and learning, and the development of multimedia and online projects. One session each year is 
reserved for teachers to travel to an eMINTS school and observe a certified eMINTS teacher. 
During both years, eISs supplement these formal professional development sessions with nine to 
10 on-site and individualized coaching sessions (about 14 hours total) and within-building 
communities of practice where teaching staff meet to share ideas, collaborate on online project 
development, and deepen their existing understanding of concepts embedded in the eMINTS 
instructional model. Finally, eMINTS provides teachers with support materials and just-in-time 
learning opportunities through online learning communities to help teachers improve their 
practice over time."^ 

Research Design: 

This study used a cluster randomized design that randomly assigned 60 high-poverty rural 
Missouri middle schools to one of three groups in Fall 2010 (Year 0). Schools assigned to Group 
1 receive the eMINTS two-year PD program (eMINTS) in 201 1- 12 and 2012-13 (Years I and 2); 
Group 2 schools receive eMINTS two-year PD in 201 1- 12 and 2012-13 (Years I and 2) plus a 
third year of Intel® Teach PD (eMINTS-i-Intel Teach) in 2013-14 (Year 3); and Group 3 schools 
conduct business as usual (BAU) with no exposure to the eMINTS or eMINTS-tlntel Teach 
(Control) in Years 1 through 3 (the duration of the study), and then receive eMINTS two-year 
PD in 2014-15 and 2015-16 (Years 4 and 5). To remove extraneous variation due to differences 
in grade configuration, randomization was blocked by the grade range in each school (Block 1 : 
P/K-8 schools; Block 2: 5-8, 6-8, 7-8 schools; Block 3: 6-12, 7-12 schools). Note that although 
the schools were randomized into three groups because of the addition of Intel Teach to the two- 
year eMINTS program in Year 3, for the Year 2 analysis, which is the focus of this report, there 
are effectively only two groups: (1) Groups 1 and 2 combined (eMINTS); and (2) Group 3 
(Control). 

Data Collection and Analysis: 

Table 1 summarizes data collection plans for student outcomes from spring 201 1 to spring 2014. 
Student outcome data includes Missouri Assessment Program (MAP) test scores in ELA and 
mathematics. Students’ 21st century skills are measured using annual 21st Century Skills 
Assessment from Learning.com (Condon, C., Dawson, M., Molefe, A., & Swanlund, A., 2009). 

A student survey is used to measure students’ engagement and academic orientation. 

Intent-to-treat estimates of impacts on student outcomes were obtained using two-level 
hierarchical regression models to adjust for the clustering of students within schools, and block 
fixed effects to control for the randomization of schools within blocks. The models also 
incorporated baseline student characteristics (prior achievement, grade, race/ethnicity, 
free/reduced price lunch status, limited English Proficiency (LEP) status, provision of an 


4 

The focus of this report is student outcomes. However, the study will also collect and answer questions related to 
teacher outcomes and fidelity measures. Teacher outcomes will be measured through annual surveys and classroom 
observations. Fidelity measures include observations of PD sessions, logs detailing in-classroom coaching activities, 
and audits of technology resources at each school site. 
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individualized educational plan (lEP), and gender); teacher characteristics (years’ teaching 
experience, degrees held, and gender); and school mean prior reading achievement. The overall 
impacts are presented as averages of district-specific impacts obtained from the regression 
models, weighted by the number of schools in each block. The mathematics and communication 
arts analytic samples (RQl) each consisted of about 3,000 students, and the analytic samples for 
21st century skills and student engagement (RQ2 and RQ3) each had about 2,500 students. 

(Please insert table 1 here.) 

Findings / Results: 

Estimated impacts after the second year of implementation on all student achievement 
outcomes — mathematics, communication arts, and 21st century skills — were not statistically 
significant. Impacts on student engagement are negative (with a 0.25 effect size) and statistically 
significant, indicating that student engagement was lower in eMINTS schools than in the control 
schools. 

Conclusions: 

As more states adopt the Common Core State Standards which aim to “provide a clear and 
consistent framework to prepare our children for college and the workforce” 

(http ://www .corestandards .org/about-the-standards) , the need to validate programs that provide 
teachers training and support to understand and use standards -based instruction and assessments 
is urgent. As one such program, the eMINTS PD program, which is gaining widespread use, 
warrants a rigorous evaluation to inform practice not only in Missouri but in all states that are 
currently implementing or are planning to implement it. 

In addition, this study will contribute rigorous empirical evidence to a limited research base on 
the integration of technology and its mediators and moderators in mathematics and English 
Eanguage Arts (Roschelle, et ah, 2010; Dynarski et ah, 2007). In particular, findings will address 
gaps in the field’s understanding how technology and student-centered learning strategies (e.g., 
problem-based learning) can be integrated in classrooms to impact students’ 21st century skills 
(Shear, Novais, and Moorthy, 2010; Shear et ah, 2009). 

It is important to note, however, that by design, the eMINTS Comprehensive Program is a two- 
year professional development program, and eMINTS -i-Intel adds a third year to the original 
program length. In this second year of implementation, teachers assigned to the eMINTS only 
and the eMINTS-i-Intel groups were offered the same comprehensive two-year eMINTS 
program; however, teachers in the eMINTS-rlntel group have not yet received a third year of 
training using the Intel Teach Program course training. As a result, this Year 2 study compares 
the group of treatment schools assigned to eMINTS or eMINTS-rlntel to the schools assigned to 
the business as usual group. According to the program developers, full program implementation 
is designed to take about three years; that is, teacher and student effects are optimized after a 
minimum of three years of implementation. Therefore, the results of the Year 2 analyses 
presented in this report should be interpreted with caution. 
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Appendix B. Tables and Figures 


Table 1: Student Data Collection Timeline 


Outcomes 

Spring 2011 
(Year 0 - 
baseline) 

Spring 2012 
(Year 1) 

Spring 2013 
(Year 2) 

Spring 2014 
(Year 3) 

MAP test 

(mathematics, 

ELA) 

X 

X 

X 

X 

Student 21st 
century 
skills test 

X 

X 

X 

X 

Student survey of 

academic 

engagement 

X 

X 

X 

X 


SREE Spring 2015 Conference Abstract Template 


B-1 



